What problem does this solve?

Loss curves improve during fine-tuning and everyone ships. But loss can drop while every output degenerates into repeating digit sequences or near-identical strings. Perplexity does not catch this — it measures token-level prediction quality, not output integrity. BLEU/ROUGE do not catch this either — they measure n-gram overlap against ground truth, not behavioral collapse against a baseline.

There is currently no metric in evaluate that catches this class of failure.

How it works

behavioral_drift composites three signals into a single drift_score (0-1, lower = worse):

1. self-BLEU — concatenated-reference BLEU-1 across fine-tuned outputs. High self-BLEU means the model is producing near-identical strings regardless of input (mode collapse).
2. digit density delta — compares the fraction of numeric characters in fine-tuned outputs vs. base model outputs. A sudden spike means the model is producing numeric garbage instead of language.
3. repetition ratio — unique-token / total-token ratio per output. Low ratio means degenerate token looping.

Each signal has a documented threshold. A diagnosis string is returned alongside the score.

Relationship to existing metrics

Not a replacement for perplexity or BLEU. An orthogonal behavioral check. Intended workflow: run drift_score alongside your standard eval suite. If perplexity improves but drift_score drops, you have a silent quality regression.

Implementation

• 147 lines of Python, single file
• Zero external dependencies beyond evaluate + datasets
• Pure deterministic computation — no model calls, no stochastic components
• Demo app (app.py) included for interactive testing via Gradio